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States are implementing some of the Perkins IV performance measures using 
different approaches and report that the greatest challenge is collecting data 
on technical skill attainment and student placement. Flexibility in Perkins IV 
and Education’s guidance permits differences in how states implement the 
measures. According to our surveys, 34 states at the secondary level and 29 at 
the postsecondary level intend to adopt Education’s recommended use of 
assessments — such as those for industry certifications — to measure technical 
skills. States reported that they face the most challenge collecting data on the 
technical skill attainment and student placement measures because of cost 
and concerns with their ability to access complete and accurate data. 

Education ensures states are implementing the Perkins IV accountability 
requirements through on-site monitoring and off-site document reviews, and 
supports states through technical assistance and guidance. Monitoring 
findings were most often related to states failing to submit complete or 
reliable data, and Education uses its findings to guide the technical assistance 
it provides to states. States reported that Education’s assistance has helped 
them implement the performance measures, but that more assistance with 
technical skill attainment would be helpful. Education is aware of states’ need 
for additional assistance and has taken actions to address this, including 
facilitating a state-led committee looking at technical assessment approaches. 

State performance measures are the primary source of data available to 
Education for determining the effectiveness of CTE programs, and Education 
relies on student outcomes reported through these measures to gauge the 
success of states’ programs. Because only 2 of 11 measures (secondary and 
postsecondary have 3 measures in common) have been implemented and 
reported on thus far, Education has little information to date on program 
outcomes. In addition, Perkins IV does not require states to report to 
Education the findings of their program evaluations. In our surveys of state 
CTE directors, nearly half of states responded that they have conducted or 
sponsored a study to examine the effectiveness of their CTE programs. We 
reviewed 7 of these studies and found that only 4 were outcome evaluations. 



Perkins IV Performance Measures 




Source: GAO analysis of Education program guidance and the Carl D. Perkins Career and Technical Education Act of 2006. 
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United States Government Accountability Office 
Washington, DC 20548 



July 29, 2009 

The Honorable Edward M. Kennedy 
Chair 

The Honorable Michael B. Enzi 
Ranking Member 

Committee on Health, Education, Labor, 
and Pensions 
United States Senate 

The Honorable Patty Murray 
Chair 

The Honorable Johnny Isakson 
Ranking Member 

Subcommittee on Employment and Workplace Safety 
Committee on Health, Education, Labor, 
and Pensions 
United States Senate 

The shift to a global economy and rapid advances in technology 
underscore the importance of preparing our current and future workforce 
for high-demand careers with 21st century skills, such as those that 
emphasize problem solving and teamwork. In the 2006-2007 program year, 1 
more than 16 million high school and college students nationwide 
participated in career and technical education (CTE) programs, which are 
designed to provide students with the academic and career and technical 
skills to help them succeed in the workforce. As authorized by the Carl D. 
Perkins Career and Technical Education Act of 2006 2 (Perkins IV), 
Congress provided states with $1.2 billion in fiscal year 2008 to support 
career and technical education in high schools and to support programs in 
postsecondary institutions, such as community colleges. The U.S. 
Department of Education (Education) estimates that approximately 5 
percent of all funds that states use for CTE programs are federal funds, 
with state and local funding generally covering the remainder. The 
American Recovery and Reinvestment Act of 2009 provides additional 
funds that states can use to help support their CTE programs. Federal 
funds for CTE programs are likely to take on increasing importance as 



'The program year generally operates from July 1 to June 30. 
2 20 U.S.C. § 2301 et seq. 
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states continue to confront mounting fiscal pressures that may lead them 
to propose cuts to secondary and postsecondary education spending used 
to support career and technical education. 

Perkins IV aims to prepare students for current or emerging high-skill, 
high- wage, or high-demand jobs by emphasizing rigorous student 
academic and technical skill achievement, increased accountability for 
student outcomes, and enhanced coordination between secondary and 
postsecondary career and technical education. It also seeks to increase 
state and local flexibility in providing career and technical education by 
involving multiple groups such as students, parents, and local 
administrators in planning and administration, and by allowing states 
flexibility in the design of their accountability systems. To increase 
accountability for student outcomes, Perkins IV established student 
performance measures at the secondary and postsecondary levels for state 
agencies, 3 such as state educational agencies or state college and 
university systems, as well as for local recipients of funds, such as school 
districts. Key performance measures include student attainment of 
academic content standards and student academic achievement standards, 
as adopted by the state in accordance with the requirements of Title I of 
the Elementary and Secondary Education Act. Overall, Perkins IV reflects 
a shift from an emphasis on vocational education — once considered by 
some to be an occupationally specific track for students with lower 
academic skills — to an emphasis on preparing students for entry into high- 
demand occupations. 

Education provides technical assistance and guidance to states regarding 
their data collection and student definitions and measurement approaches. 
States report annually to Education on their progress in meeting their 
performance targets for the measures. In light of a governmentwide focus 
on performance and accountability, you asked us to examine (1) how 
states have implemented the Perkins IV performance measures, and what, 
if any, challenges they have faced in implementing the measures; (2) to 
what extent Education has ensured that states are implementing the new 
performance measures and supported states in their efforts; and (3) what 
Education knows about the effectiveness of CTE programs. 

To answer our three research questions, we collected data through 
multiple methods. First, to gather state-level information on Perkins IV 



^Throughout this report we refer to state agencies as “states.” 
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implementation, we collected information through two Web-based surveys 
of state CTE directors, at the secondary and postsecondary levels, in the 
50 states and the District of Columbia. The surveys obtained information 
on the types of data states collect for the student performance measures 
and challenges they face; technical assistance, guidance, and monitoring 
states receive from Education; and how states evaluate their CTE 
programs. We administered the surveys between January and April 2009 
and received responses from all 50 states and the District of Columbia. 
While we did not fully validate specific information that state officials 
reported through our surveys, we reviewed the information to determine 
that their responses were complete and reasonable and found the 
information to be sufficiently reliable for the purposes of this report. This 
report does not contain all the results from the surveys. The surveys and a 
more complete tabulation of the results can be viewed online at 
GAO-09-737SP. In addition to our surveys, we collected information from 
site visits to California, Minnesota, and Washington state. These states 
represent variation across characteristics such as the type of state agency 
eligible to receive Perkins funds; amount of Perkins IV funds received in 
fiscal year 2008; and type of approach used to assess how students attain 
technical skills, a key program outcome. We interviewed secondary and 
postsecondary officials at the state level and officials from local recipients 
of Perkins funds — that is, school districts and postsecondary 
institutions — that varied by number of CTE students served, amount of 
Perkins funding received, and geographic location (urban versus rural). 

We also reviewed relevant federal legislation and agency guidance and 
interviewed Education officials to obtain information on how states have 
implemented the performance measures, how Education has monitored 
and supported states in their efforts to implement the performance 
measures, and what Education knows about how states are evaluating 
their local CTE programs. To analyze how states are evaluating CTE 
programs, we reviewed state Perkins plans and annual reports submitted 
to Education from the 50 states and the District of Columbia. See appendix 
I for detailed information on our surveys and site visits. 

We conducted this performance audit from August 2008 to July 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Background 



Under Perkins IV, 
Education Allocates Funds 
for Career and Technical 
Education to States in 
order to Improve Local 
CTE Programs 



The principal source of federal funding for CTE, Perkins IV authorizes 
federal grant funds for the enhancement of CTE for secondary and 
postsecondary students. In fiscal year 2008, Congress appropriated $1.2 
billion for the improvement of local CTE programs. Education’s Office of 
Vocational and Adult Education allocates the funds to states, 4 which retain 
up to 15 percent of the funds for administration and state leadership of 
CTE programs, 5 before passing at least 85 percent of the funds on to local 
recipients of funds, such as local school districts and community colleges. 
States determine the percentage of funds that will be allocated to the 
secondary and postsecondary levels. The majority of funds allocated to the 
secondary level are passed on to local recipients based on the school 
district’s share of students from families below the poverty level for the 
preceding fiscal year. Postsecondary funds are primarily allocated based 
on the institution’s share of Pell Grant recipients. 6 



Perkins IV Established 
Performance Measures for 
Secondary and 
Postsecondary Levels and 
Requires States and Local 
Recipients to Report on 
Program Outcomes 



Perkins IV established six student performance measures at the secondary 
level and five performance measures at the postsecondary level. These 
measures represent a range of student outcomes, such as attainment of 
technical skills and placement in employment or further education 
following the completion of CTE programs. In addition, the measures 
include the nontraditional participation and completion of students from 
an underrepresented gender in programs with significant gender 
disparities (such as women participating in auto repair), among others 
(see tables 1 and 2 for a description of the Perkins IV performance 
measures). To ease states’ transition to the new provisions in Perkins IV, 



Education initially allocates funds to states based on the population of three age groups 
(15 to 19, 20 to 24, and 25 to 65) and the state’s per capita income. If a state’s allocation 
exceeds the amount allocated during fiscal year 2006, a new formula is used to allocate 
funds at the state level. 

Administration funds may be used by states to support the development of the state 
Perkins plans, review of the local plans, monitoring and evaluation of program 
effectiveness, ensuring compliance with federal laws, technical assistance, and supporting 
and developing state data systems. State leadership funds must be used by the states to 
improve CTE programs through nine different activities, including the assessment of CTE 
programs and expanding the use of technology in CTE programs. 

The Federal Pell Grant Program provides need-based grants to low-income undergraduate 
and certain postbaccalaureate students to promote access to postsecondary education. 
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Education permitted states to submit a 1-year transition plan that covered 
only the first program year of Perkins IV implementation, 2007-2008. 
Accordingly, states were required only to implement and report 
performance on two secondary performance measures for the 2007-2008 
program year: academic attainment and student graduation rates. These 
two measures are based on the same academic attainment and student 
graduation rate measures required by Title I of the Elementary and 
Secondary Education Act. Beginning in the 2008-2009 program year, states 
are required to report on student outcomes for all of the performance 
measures. States will report these outcomes to Education in December 
2009. 



Table 1 : Perkins IV Performance Measures at the Secondary Level 


Performance measure 


Description 


Program year 
implemented 


Academic attainment in 
reading/language arts and 
mathematics 


Student attainment of challenging academic content and academic 
achievement standards, adopted from Title 1 of the Elementary and 
Secondary Education Act. 


2007-2008 


Technical skill attainment 


Student attainment of career and technical skill proficiencies, 
including achievement on technical assessments aligned with 
industry-recognized standards, if available and appropriate 


2008-2009 


Secondary school completion 


Student rates of attainment of a secondary school diploma; a 
General Educational Development (GED) credential or equivalent; 
and proficiency credential, certificate, or degree, in conjunction with 
a secondary school diploma 


2008-2009 


Student graduation rate 


Student graduation rates, as described in Title 1 of the Elementary 
and Secondary Education Act 


2007-2008 


Student placement 


Student placement in postsecondary education or advanced 
training, military service, or employment 


2008-2009 


Nontraditional participation and 
completion 


Student participation in, and completion of, CTE programs that lead 
to nontraditional fields, such as women in automotive programs or 
men in child development 


2008-2009 



Source: GAO analysis of Education program guidance and the Carl D. Perkins Career and Technical Education Act of 2006. 

Note: States report to Education on attainment in reading and language arts as two separate 
measures, consistent with how states report on academic achievement under Title I of the 
Elementary and Secondary Education Act. 
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Table 2: Perkins IV Performance Measures at the Postsecondary Level 


Performance measure 


Description 


Program year 
implemented 


Technical skill attainment 


Student attainment of challenging career and technical skill 
proficiencies, including achievement on technical assessments 
aligned with industry-recognized standards, if available and 
appropriate 


2008-2009 


Credential, certificate, or degree 
attainment 


Student attainment of an industry-recognized credential, a 
certificate, or a degree 


2008-2009 


Student retention or transfer 


Student retention in postsecondary education or transfer to a 
baccalaureate degree program 


2008-2009 


Student placement 


Student placement in military service or apprenticeship programs, or 
placement or retention in employment 


2008-2009 


Nontraditional participation and 
completion 


Student participation in, and completion of, CTE programs that lead 
to employment in nontraditional fields, such as women in 
automotive programs or men in child development 


2008-2009 



Source: GAO analysis of Education program guidance and the Carl D. Perkins Career and Technical Education Improvement Act of 
2006 . 



Perkins IV requires states to negotiate specific performance targets with 
Education and to annually report their performance to Education. It also 
requires local recipients to negotiate performance targets with the states 
and to annually report to the state their progress toward meeting these 
targets. Perkins IV established additional accountability requirements for 
states and local recipients, including actions to address states that do not 
meet all of their performance targets. Under Perkins IV, if a state does not 
meet at least 90 percent of its targets for one or more of the performance 
measures, it is required to develop and implement a program improvement 
plan that describes how it will address its failing performance targets. 

Prior to Perkins IV, states were only required to develop and implement a 
program improvement plan if they failed to meet their targets in all of their 
performance measures, not just one measure. States can also face 
financial sanctions. For example, Education can withhold all or a portion 
of funds if a state does not implement a program improvement plan, show 
improvement in meeting its failing performance measure, or meet the 
target for the same performance measure for 3 consecutive years. 7 Local 
recipients that do not meet at least 90 percent of their performance targets 
have the same program improvement requirements as the state and face 
similar sanctions from the state. In the event of financial sanctions, 
Education is required to use the withheld funds to provide technical 



Education officials told us that, to date, they have not withheld funds from any states. 
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assistance to the state for improving its performance on the measures and 
the state is to use funds withheld from local recipients to provide CTE 
services and activities to students. 



Education Developed 
Nonregulatory Guidance to 
Assist States with Perkins 

IV 



In order to implement the performance measurement requirements of 
Perkins IV, states must define which students will be included in the 
measures and collect data for each of the performance measures at the 
secondary and postsecondary levels. For example, states define the 
minimum requirements, such as a certain number of CTE credits that a 
student would need to obtain in order to be identified as a student 
concentrating in CTE. Education has taken a range of actions to help 
states with these activities. For example, in January 2007, Education began 
issuing nonregulatory guidance to states to help them develop their 
student definitions and data collection approaches for the performance 
measures. 8 Education also issued guidance to states on the information 
states must include in their state Perkins plans and in the annual reports 
that they submit to Education. In the state plans, states must detail how 
they intend to implement the performance measures, and in the annual 
reports states must describe their progress in meeting the negotiated 
performance targets. 9 



In Addition to 
Implementing the Perkins 
Performance Measures, 
Perkins IV Also Requires 
States to Annually 
Evaluate Their Local CTE 
Programs 



In addition to implementing performance measures, states are required to 
evaluate programs, services, and activities supported with Perkins funds 
and to report to Education in their state plans how they intend to conduct 
these evaluations. To meet this requirement, states describe the 
approaches, such as the use of state-developed standards, they will use to 
evaluate local CTE programs. In addition, Education requires states to 
include a description of how they used Perkins funds to evaluate their 
local CTE programs in their annual reports. 



8 Section 318 of the Carl D. Perkins Career and Technical Education Act of 2006 provides 
that the Secretary of Education may issue regulations only to the extent necessary to 
administer and ensure compliance with the specific requirements of the act. 

9 Perkins IV requires each state receiving Perkins funds to submit to Education a state plan 
describing how the state will meet the requirements of the act. We refer to these plans as 
“state Perkins plans.” States are also required by Perkins IV to submit an annual report that 
Education refers to as a “Consolidated Annual Report” and we refer to as an “annual 
report.” 
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States Are 
Implementing Some 
Performance 
Measures Using 
Different Approaches 
and Report That the 
Greatest Challenge Is 
Collecting Data on 
Technical Skill 
Attainment and 
Student Placement 

A key feature of Perkins IV — to enhance state and local flexibility in 
developing, implementing, and improving career and technical 
education — allows for considerable variation in how states implement 
some performance measures. 10 While Perkins IV was designed to 
strengthen accountability for results at the state and local levels, it also 
allows states to establish their own accountability systems, including their 
own data collection methods for the performance measures. Of the 11 
performance measures, the secondary and postsecondary levels have 3 
measures in common: technical skill attainment, student placement, and 
participation in and completion of nontraditional programs (see fig. I). 11 
States may also include additional, state-developed performance measures 
in their accountability systems. For example, Washington state added 
three performance measures — earnings, employer satisfaction, and CTE 
student satisfaction — to its accountability system. 



Flexibility in Law and 
Guidance Allows for 
Differences in How States 
Implement Some 
Performance Measures 
and Results in Variation in 
the Student Outcome Data 
Education Will Collect 



10 Section 2 of the Carl D. Perkins Career and Technical Education Act of 2006. 

^Slight variations exist in the definitions for the technical skill attainment and student 
placement measures at the secondary and postsecondary levels. 
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Figure 1 : Perkins IV Performance Measures at the Secondary and Postsecondary 
Levels 




Source: GAO analysis of Education program guidance and the Carl D. Perkins Career and Technical Education Act of 2006. 



Consistent with Perkins IV, Education’s guidance to states also allows for 
flexibility. Education issued nonregulatory guidance that proposed 
specific definitions that could be adopted by states to develop each of the 
secondary and postsecondary performance measures. It also identified 
preferred approaches for collecting data for certain measures such as 
student technical skill attainment. However, Education noted that in 
accordance with Perkins IV, states could propose other definitions and 
approaches to collect data for the required performance measures if they 
meet the requirements of the law. 

We found through our surveys of state CTE directors that states vary 
considerably in the extent to which they plan to follow Education’s 
guidance — specifically with regard to the technical skill attainment and 
secondary school completion measures. As a result, Education will collect 
student outcome data that vary across states for the same measures. This 
can create challenges for Education to aggregate student outcomes at the 
national level. For example, a majority of states reported that they will use 



Page 9 



GAO-09-683 Career and Technical Education 



technical assessments — the approach recommended in Education’s 
guidance — to measure student attainment of skills at the secondary and 
postsecondary levels. These include assessments leading to industry-based 
certificates or state licenses. However, a number of states will rely on 
other approaches to collect data for the performance measure, including 
grade point average (GPA), program completion, or other methods (see 
table 3). 



Table 3: Approaches That States Will Use to Collect Data on the Student Technical 
Skill Attainment Measure, by Number of States 


Approach 


Secondary level 


Postsecondary level 


Technical assessments 


34 


29 


Grade point average 


8 


17 


Program completion 


13 


13 


Other methods 


10 


9 



Source: GAO analysis of secondary and postsecondary surveys of state CTE directors. 

Note: States may use more than one data collection method. 



Officials in the states we visited provided a variety of reasons for their use 
of alternate methods to measure students’ attainment of technical skil l s. 
For example, postsecondary state officials in California said that a CTE 
instructor’s overall evaluation of a student’s technical skill proficiency, in 
the form of a final grade, is a better measure of technical skill attainment 
than third-party technical assessments, and can more effectively lead to 
program improvement. They questioned the value of technical 
assessments, in part because assessments often cannot keep pace with 
technology and changing CTE program curricula, such as curricula for 
digital animation. A Washington state official told us that the state plans to 
use program completion to measure technical skills at the postsecondary 
level, noting that each postsecondary CTE program incorporates industry- 
recognized standards into the curriculum. He noted that a national system 
of third-party assessments may not be adequate or appropriate, because it 
would not necessarily incorporate the same standards. Local school 
officials in Minnesota said that they will report on CTE course completion 
for this measure. Because CTE courses undergo curriculum review by 
teachers as well as industry advisors, and align with relevant 
postsecondary programs in the area, school officials told us course 
completion is sufficient to satisfy the definition of technical skill 
attainment. 

Education’s guidance also allows for considerable variation in the types of 
technical assessments states can use and when they can administer them. 
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Most states at the secondary level reported in our survey that they plan to 
use industry-developed certificates or credentials most often administered 
at the end of a program, such as a certificate awarded for an automotive 
technician. At the postsecondary level, states plan to most often rely upon 
the results of assessments for state licenses, such as state nursing licenses, 
to measure technical skills (see fig. 2). 



Figure 2: Number of States Planning to Use Technical Assessments Administered 
at Various Times, by Type of Assessment 





Assessment for industry-developed certificate or credential 

Assessment for state license 

Nationally developed assessment 

State-developed assessment 

Locally developed assessment 



Source: GAO analysis of secondary and postsecondary surveys of state CTE directors. 



Note: A state may administer technical assessments at different times to CTE students. For example, 
assessments can follow the completion of a CTE course or program. This figure includes all states 
that reported that they will administer technical assessments at various times. 

However, we found that while a majority of states plan to use assessments 
to report to Education, the assessments are not currently in widespread 
use. For example, more than half of states at the secondary and 
postsecondary levels reported that they plan to use these assessments to 
report on few to none of their state-approved CTE programs in the 2008- 
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2009 program year. Some states at the secondary level reported that they 
will use a combination of methods — including GPA or program 
completion — to report on technical skill attainment. 

We also found that states differ in whether they plan to report student data 
on GED credentials, part of the secondary school completion measure. 12 
Thirty states reported through our survey that they do not plan to report 
GED data to Education for the 2008-2009 program year, while 18 reported 
that they would. About one-third of all states cited their ability to access 
accurate GED data as a great or very great challenge. For example, state 
officials we interviewed said states face difficulty tracking the students 
that leave secondary education and return, sometimes several years later, 
to earn a GED credential. An Education official said that the agency is 
aware of the challenges and limitations states face in collecting GED data 
and that the agency may need to provide technical assistance to states on 
ways to collect these data. 



States Face the Most 
Challenge Collecting Data 
on Student Technical Skill 
Attainment and Placement 
Measures because of Cost 
and Data Concerns 



States reported in our surveys that they face the most difficulty in 
collecting student data for two of the performance measures: technical 
skill attainment and student placement (see fig. 3 and fig. 4). Thirty-eight 
states at the secondary level reported that they face great or very great 
challenges in collecting data on student technical skill attainment, while, 
similarly, 14 said they face challenges collecting data on student 
placement. The results were similar at the postsecondary level: 39 states 
reported great or very great challenges with the technical skill attainment 
measure and 11 cited a similar level of difficulty with student placement. 



12 Although the collection of GED data is required by Perkins IV and included as part of 
Education’s guidance, an Education official said that the agency approved all state Perkins 
plans despite the fact that some states would not be able to accurately track and report 
GED attainment data. The official said that the lack of GED data will be addressed in future 
state monitoring and auditing visits. 
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Figure 3: Number of States Reporting Data Collection for Perkins Performance 
Measures as a Great or Very Great Challenge at the Secondary Level, by 
Performance Measure 




0 10 20 30 40 

States 

Source: GAO analysis of secondary surveys of state CTE directors. 



Figure 4: Number of States Reporting Data Collection for Perkins Performance 
Measures as a Great or Very Great Challenge at the Postsecondary Level, by 
Performance Measure 
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Source: GAO analysis of postsecondary surveys of state CTE directors. 
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States reported that the technical skill attainment measure at the 
secondary and postsecondary levels was most challenging to implement 
because of costs and the states’ ability to collect accurate and complete 
student data. Specifically, states reported that the costs of state-developed 
assessments and third-party technical assessments — such as those for 
industry certifications — are high and often too expensive for many 
districts, institutions, or students. 13 Several state CTE directors 
commented in our surveys that their Perkins funds are inadequate to pay 
for these assessments and additional funds would be necessary to cover 
the costs. Another CTE director stated that economically disadvantaged 
students cannot afford the cost of assessments. In addition to challenges 
due to cost, states are limited in their ability to access accurate and 
complete data. For example, a state official said that Washington state 
does not have data-sharing agreements with assessment providers to 
receive the results of student assessments. As a result, the state will have 
to rely largely on students to self-report the results of their assessments, 
which raises concerns of data quality. Challenges such as these likely 
contribute to some states’ use of other data — such as GPA or program 
completion — to collect and report information for this key student 
performance measure. 

Some states also reported difficulty collecting data on CTE students after 
they leave the school system. States at the secondary and postsecondary 
levels reported that their greatest challenge with the student placement 
measure is collecting data on students that are employed out of state. As 
we previously reported, state wage records, such as Unemployment 
Insurance data, track employment-related outcomes only within a state, 
not across states. 14 A number of states commented in our surveys on 
challenges in tracking students because of the lack of data sharing across 



13 For example, Cisco computer-based certification exams generally range from $80 to $325. 
Some certifications, however, may cost as much as $1,400. 

14 Each state maintains Unemployment Insurance wage records to support the process of 
providing unemployment compensation to unemployed workers. The records are compiled 
from data submitted to the state each quarter by employers and primarily include 
information on the total amount of income earned during that quarter by each of their 
employees. 
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states. 15 We found that states face challenges in tracking students 
employed out of state regardless of the method they most commonly use 
to collect student placement data. Thirty-eight states at the secondary 
level will use student survey data from the state, school district, or a third 
party to track student placement and report to Education, while 41 states 
at the postsecondary level will rely on state wage record data, despite 
potential gaps in student data (see fig. 5). 



Figure 5: Most Commonly Used Methods to Collect Student Placement Data, by 
Number of States and Educational Level 



Type of data used 



States 




41 



Secondary level 
Postsecondary level 



50 



Source: GAO analysis of secondary and postsecondary surveys of state CTE directors. 



Note: States may use more than one data collection method. 



States also cited other challenges in obtaining data on student placement 
for CTE students. At the secondary level, states reported that their next 
greatest challenge is linking secondary and postsecondary data systems in 
order to track students that pursue higher education after graduation. To 
help overcome this challenge, Minnesota — one of the states we visited — 
recently passed legislation to allow data sharing between the secondary 



15 An Education official said that the agency provides information to states about various 
potential sources of student placement data, including the Wage Record Interchange 
System. This system, a Department of Labor initiative, was developed to facilitate the 
exchange of wage data between participating states for the purpose of assessing and 
reporting on state and local performance for programs authorized under the Workforce 
Investment Act of 1998. 
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and postsecondary levels. 16 Our survey also found that states’ next greatest 
challenge at the postsecondary level was collecting data on students who 
are self-employed after leaving postsecondary institutions. Community 
college officials in California said that while they rely on Unemployment 
Insurance wage record data, the data are incomplete and do not capture 
information on the self-employed, a group that is important for the 
measurement of CTE outcomes at the postsecondary level. 

States face similar challenges of cost and ability to access accurate data 
for the remaining performance measures. For example, states at the 
secondary level commented on data challenges for the academic 
attainment and student graduation rate measures. 17 Specifically, several 
states cited problems in obtaining data from separate student data systems 
containing academic and CTE information. This can be particularly 
challenging for states that are trying to match student data from different 
systems in order to track required CTE student outcomes. In addition, at 
the postsecondary level, states cited challenges in tracking student 
retention in postsecondary education or student transfer to a 
baccalaureate degree program. In particular, accessing student data from 
out-of-state and private institutions and the high costs required to track 
these students were identified as the most challenging issues. States most 
often reported that they will track these students through their state 
postsecondary data systems. 



lb Under the Minnesota law, the following educational data may be shared between the state 
educational agency and the state office of higher education for improvement purposes: 
attendance data, including name of school or institution, school district, year or term of 
attendance, and term type; student demographic and enrollment data; academic 
performance and testing data; and special academic services received by a student. 
However, data may be analyzed or reported only in the aggregate. 

17 In December 2008, states were required to report only on the student academic 
attainment and graduation rate measures as required under Title I of the Elementary and 
Secondary Education Act for the 2007-2008 program year. 
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Education Uses Risk- 
Based Monitoring to 
Ensure 

Implementation of the 
Performance 
Measures and 
Supports States 
through Technical 
Assistance and 
Guidance 

As we have previously reported, effective monitoring is a critical 
component of grant management. The Domestic Working Group’s 
suggested grant practices state that financial and performance monitoring 
is important to ensure accountability and attainment of performance 
goals . 18 Additionally, GAO recently reported on the importance of using a 
risk-based strategy to monitor grants and noted that it is important to 
identify, prioritize, and manage potential at-risk grant recipients, given the 
large number of grants awarded by federal agencies . 19 Education’s 
approach to monitoring Perkins is consistent with these suggested grant 
practices. According to its Perkins monitoring plan, Education selects 
which states to monitor based on a combination of risk factors and 
monitors states in two ways: through on-site visits and off-site reviews of 
state plans, budgets, and annual reports for those states not visited in a 



Education Uses Risk- 
Based Monitoring and 
Reviews State Annual 
Reports to Ensure 
Implementation of the 
Performance Measures 



18 The Domestic Working Group Grant Accountability Project, Guide to Opportunities for 
Improving Grant Accountability, October 2005. The group was composed of 
representatives from federal, state, and local audit organizations and is chaired by the 
Comptroller General of the United States. 

19 See GAO-08-486. 
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given year. 20 To determine which states it will visit for on-site monitoring, 
Education uses a combination of risk factors, such as grant award size, 
issues identified through reviews of state Perkins plans, and time elapsed 
since Education’s last monitoring visit. Education officials told us that 
their goal is to visit each state at least once every 5 years and reported that 
they have conducted on-site monitoring visits to 28 states since 2006. 
Education officials also told us that the same monitoring team performs 
both on-site and off-site reviews, which officials said helps to ensure 
continuity between the reviews. Furthermore, when conducting the off- 
site reviews, the monitoring team looks for trends in state data and for any 
problems with state data validity and reliability. The team uses a checklist 
to match performance data to the data states report in their required 
annual reports. 

According to Education’s inventory of open monitoring findings, as of May 
2009, 9 of the 28 open findings were related to accountability and states 
failing to submit complete or reliable data. For example, in a February 
2008 monitoring visit, Education found that the monitored state’s data 
system had design limitations that affected the state’s ability to collect and 
assess data on career and technical education students. Specifically, 
Education found that the various data systems across the local secondary 
and postsecondary levels did not share data with the state-level CTE data 
system. This data-sharing issue raised doubts about the validity and 
reliability of the state’s Perkins data. Education tracks the findings from 
each state’s monitoring visit in a database and reviews the findings in an 
internal report that is updated monthly. Additionally, if a state has open 
findings, the state may be required to report corrective actions to 
Education in the state’s annual report. Officials told us that the amount of 
time it takes for a state to close out a finding depends upon the nature of 
the finding. For example, a finding related to accountability may take up to 
a year to resolve because a state may have to undertake extensive actions 
to address the deficiency. Education officials reported that their 
monitoring process emphasizes program improvement rather than 



20 According to Education’s fiscal year 2009 monitoring plan, full visits are weeklong, on-site 
reviews that address compliance in seven areas, including accountability, state or program 
administration, fiscal program responsibility, and programs of study. Targeted visits are 2- 
day, on-site reviews that address one or more of the seven areas, depending on the issues 
and needs of the state. An Education official told us that it is typical for some states to 
receive several targeted reviews before receiving a full review. Education officials also told 
us that targeted reviews can be used to follow up on a state’s progress implementing 
corrective actions following a full monitoring review. 
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focusing solely on compliance issues and that they use monitoring findings 
to guide the technical assistance they provide to the states. 

To evaluate its monitoring process, Education sends a survey to the CTE 
directors of states that were monitored that year and asks them to rate the 
format and content of Education’s Perkins monitoring process. For 
example, the survey asks states to report on whether they received 
sufficient notice that the site visit was going to take place, whether the 
monitoring team provided on-site technical assistance, and whether the 
state received a written report within a reasonable time frame following 
the visit. We reviewed Education’s summaries of the state surveys and 
found that for 2004 and 2005, the results of these surveys were generally 
positive. For example, in a 2004 monitoring evaluation report, the 10 states 
that were surveyed all reported that they had received sufficient notice 
about the monitoring visit and that Education staff provided on-site 
technical assistance. According to our survey of secondary-level CTE 
directors, about half of states have had a monitoring visit within the last 3 
years, and almost all of the states whose monitoring visit resulted in 
findings said that Education worked with them to ensure that the findings 
were addressed. 



Education Supports States Education provides states with guidance, technical assistance, and a 
by Providing Technical variety of other resources and is taking actions to meet states’ need for 

Assistance and Guidance additional help. Since Perkins IV was enacted, Education has issued 

guidance to states on topics such as instructions for developing the state 
Perkins plans and annual reports, as well as guidance related to the 
performance measures. For example, Education’s guidance provides 
clarification to states on what information each state has to submit to 
Education before it can receive its grant award for the next program year, 
such as any revisions a state wants to make to its definitions of student 
populations, measurement approaches, and proposed performance levels 
for each of the measures. Some of the guidance resulted from Education’s 
collaborative efforts with states. For example, Education’s guidance to 
states on student definitions and measurement approaches incorporated 
the input given by state CTE directors during national conference calls 
between states and Education. Other guidance addresses questions raised 
by states during national Perkins IV meetings, such as how a state should 
negotiate performance levels with its local recipients. 

In addition to guidance, Education offers states technical assistance from 
Education staff — called Regional Accountability Specialists — and through 
a private contractor. Education officials told us that each Regional 
Accountability Specialist works with a specific group of states to negotiate 
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state data collection approaches for the performance measures. In 
addition, each specialist maintains regular contact with his or her states 
throughout the year and provides assistance on other issues, such as 
reporting requirements and program improvement plans. In addition to the 
Regional Accountability Specialists, Education also provides states with 
technical assistance by using MPR Associates, a private contractor. 21 MPR 
Associates provides technical assistance that generally includes on-site 
visits and follow-up discussions to help states improve their CTE programs 
and facilitate data collection for the performance measures. For example, 
MPR Associates met with one state to assist with developing population 
definitions and measurement approaches that aligned with Education’s 
guidance and helped another state with developing a plan for 
implementing secondary and postsecondary technical skill assessments. 
After providing technical assistance to a state, MPR Associates develops a 
summary report, which is then published on Education’s information- 
sharing Web site, the Peer Collaborative Resource Network. Education 
also offers states a range of other resources, including data work groups 
and monthly conference calls. See table 4 for a description of the various 
ways in which Education provides assistance to states. 



Table 4: Education’s Assistance to States for Perkins IV Implementation 



Form of assistance 


Description 


Data Quality Institute 


The Data Quality Institute is an Education-hosted seminar that helps states improve the 
quality and consistency of the data states use to report on the Perkins performance 
indicators. Education has hosted 15 seminars that have included participation by state 
officials at the secondary and postsecondary levels. Seminars typically focus on a particular 
issue, such as measuring technical skill attainment. Since 2000, the seminars have been 
held once a year for 2 days. The 2009 seminar was held via the Internet and included 460 
participants representing 50 states. 


Next Steps Work Group 


The Next Steps Work Group is primarily composed of Education accountability staff and 
state CTE directors and their accountability staff. Education officials told us that the work 
group is one of their primary communication tools for working with state data contacts. The 
Next Steps Work Group also has subgroups based on the larger group’s interests. Current 
subgroups are focusing on technical skill assessments, data disaggregation, and the 
consistency of certain performance measures. 


Peer Collaborative Resource Network 


The Peer Collaborative Resource Network is a resource- and information-sharing forum for 
state CTE professionals. It serves as a peer-to-peer forum for states to improve Perkins IV 
implementation and data quality, as well as providing information on other Education grant 
programs and national initiatives related to career and technical education. 



2 1 MPR Associates provides technical assistance to individual states as they implement 
Perkins IV and will have worked with about 20 states between 2007 and December 2009. 
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Form of assistance 


Description 


State directors conference calls 


Education hosts periodic conference calls and Web-based seminars with state CTE 
directors, during which technical assistance and guidance are provided. Call topics have 
included technical skill assessments, recaps of national CTE policy meetings, and state 
presentations on past experiences with customized technical assistance. 


Attendance at national conferences 


Education officials have conducted workshops and presented at a number of national 
conferences. These include conferences of the Association of Career and Technical 
Education and the National Association of State Directors of Career and Technical 
Education Consortium. 


CTE Research 


Through the National Center for Research in Career and Technical Education, the 
Department of Education supports research and evaluation, development, dissemination, 
technical assistance and training activities, as well as other activities aimed at improving 
CTE education. 



Source: GAO analysis of Education documents and interviews with Education officials. 



Most states reported that the assistance provided by Education has helped 
them implement the performance measures, but that more assistance in 
the area of technical skill attainment would be helpful. In our survey, 
states responded positively about their Regional Accountability Specialist 
and all of Education’s other forms of assistance, including the Data Quality 
Institute and the Next Steps Work Group. States also reported that more 
nonregulatory guidance and more individual technical assistance would 
improve their ability to implement the performance measures. Of the 
states that provided additional information on the areas in which they 
wanted assistance, 4 of 16 states at the secondary level and 9 of 20 states 
at the postsecondary level said that they wanted assistance on the 
technical skill attainment measure. Specifically, some of the states that 
provided additional information said they would like Education to clarify 
its expectations for this measure, to provide states with a library of 
technical assessments, and to provide state-specific assistance with 
developing low-cost, effective technical assessments. States also raised 
issues regarding the performance measures and their state’s data 
collection challenges. For example, one state reported that it was unsure 
how a state should report technical skill attainment as a single measure for 
over 400 distinct CTE programs. 

We found that Education officials were aware of states’ need for additional 
assistance and that Education has taken some actions to address these 
needs, particularly in the area of technical assessments. For example, 
through the Next Steps Work Group, Education facilitated a technical 
skills attainment subgroup that is led by state officials and a national 
research organization. The subgroup reviewed state Perkins plans and 
annual reports for technical skill assessment strategies that states 
reported to Education for consideration in upcoming guidance. Education 
also collaborated with MPR Associates to conduct a study on the 



Page 21 



GAO-09-683 Career and Technical Education 





feasibility of a national technical assessment clearinghouse and test item 
bank. 22 The study, conducted with several CTE research organizations and 
state-level consortia, proposed national clearinghouse models for 
technical assessments. MPR Associates concluded that clarifying 
ownership, such as who is responsible for the development and 
management of the system, and securing start-up funding were the two 
most likely impediments to creating such a system. The report was 
presented to states at the October 2008 Data Quality Institute seminar, and 
Education officials reported that they are working with organizations such 
as the National Association for State Directors of Career and Technical 
Education Consortium and the Council of Chief State School Officers to 
implement next steps. 

In addition to helping states with the technical skill attainment measure, 
Education also has taken actions to improve its information-sharing Web 
site, the Peer Collaborative Resource Network. Specifically, a Next Steps 
Work Group subcommittee surveyed states for suggested ways to improve 
the Web site and reported that states wanted to see the information on the 
site kept more current. The subcommittee reported in December 2008 that 
Education would use the survey results to develop a work plan to update 
the Web site. In May 2009, Education officials reported that they had 
implemented the work plan and were piloting the revamped site with 
selected state CTE directors before the department finalizes and formally 
launches the site. 



Education Relies on 
the Performance 
Measures to Gauge 
the Success of State 
CTE Programs 



State performance measures are the primary source of data available to 
Education for determining the effectiveness of CTE programs, and 
Education relies on student outcomes reported through these measures to 
gauge the success of states’ programs. While Perkins IV requires states to 
evaluate their programs supported with Perkins funds, it only requires 
states to report to Education — through their state plans — how they intend 
to evaluate the effectiveness of their CTE programs. It does not require 
states to report on the findings of their evaluations and does not provide 
any specific guidance on how states should evaluate their programs. 



22 According to the MPR Associates’ study, a test item bank contains questions submitted by 
various business, industry, and education sources and an assessment clearinghouse 
contains information about industry-recognized national assessments that may be adopted 
or adapted for use. 
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Because only 2 of 11 measures have been implemented and reported on 
thus far, Education has little information to date on program outcomes. In 
program year 2007-2008, Education required states to implement and 
report only the academic skill attainment and graduation rate measures. 
States are required to provide Education with outcome data for the 
remaining 9 secondary and postsecondary measures in December 2009. 
According to Education’s annual report for program year 2007-2008, 43 
states met their targets for the academic attainment in reading/language 
arts measure, 38 states met their targets for the academic attainment in 
mathematics measure, and 46 states met their targets for the graduation 
rate measure. 23 

We analyzed the state plans of all 50 states and the District of Columbia 
and found that, as required by Perkins IV, states provide a description to 
Education on how they are evaluating their CTE programs. 24 The type of 
information that states provided varied. For example, some states 
described the databases they use to capture key data and others explained 
how they use state-developed performance measures to evaluate their 
programs. Perkins IV does not require that states include information on 
what their evaluations may have found in terms of the success of a 
program. In our surveys of state CTE directors, nearly half of states (23 
states at the secondary level and 21 states at the postsecondary level) 
responded that they have conducted or sponsored a study, in the past 5 
years, to examine the effectiveness of their CTE programs. In response to 
these survey results, we collected seven studies that states identified as 
evaluations of their program effectiveness. We developed an instrument 
for evaluating these studies and determined the type of evaluation and 
methodology used by states in these studies. We determined that four of 
the studies were outcome evaluations and the remaining three studies 



23 Student outcome data for these performance measures are collected and reported by the 
local recipients to the state. The state reports this information to Education. 

^Education’s instructions to states mirror the language in Perkins IV. The instructions ask 
states to describe in their state plans how the eligible agency will annually evaluate the 
effectiveness of its career and technical education programs, and describe, to the extent 
practicable, how the state is coordinating such programs to ensure nonduplication with 
other federal programs. 
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were not outcome, impact, or process evaluations . 25 For example, one 
state found in its outcome evaluation that high school graduates who 
completed a CTE program of study entered postsecondary institutions 
directly after high school at the same rate as all graduates. 



Concluding 

Observations 



Perkins IV provides states with considerable flexibility in how they 
implement the required performance measures and how they evaluate the 
effectiveness of their CTE programs. While this flexibility enables states to 
structure and evaluate their programs in ways that work best for them, it 
may hinder Education’s ability to gain a broader perspective on the 
success of state CTE programs. Specifically, differences in how states 
collect data for some performance measures may challenge Education’s 
ability to aggregate student outcomes at a national level and compare 
student outcomes on a state-by-state basis. Further, Education is limited in 
what it knows about the effectiveness of state CTE programs, beyond 
what states report through the performance measures. Perkins only 
requires that states report on how they are evaluating their programs, and 
does not provide any guidance on how states should evaluate their 
programs or require that states report on the outcomes of their 
evaluations. Education is working with states to help them overcome 
challenges they face in collecting and reporting student outcomes, and 
over time, states may collect more consistent data for measures such as 
technical skill attainment. As states become more adept at implementing 
the Perkins performance measures, they will be better positioned to 
conduct more rigorous evaluations of their CTE programs. However this 
information may not be reported to Education. If policymakers are 
interested in obtaining information on state evaluations, they will need to 
weigh the benefits of Education obtaining this information with the 
burden of additional reporting requirements. 



26 GAO, Performance Measurement and Evaluation: Definitions and Relationships, 
GAO-05-739SP (Washington, D.C.: May, 2005). This product explains three principal types of 
program evaluation. An outcome evaluation assesses the extent to which a program 
achieves its outcome-oriented objectives. It focuses on outputs and outcomes (including 
unintended effects) to judge program effectiveness but may also assess program process to 
understand how outcomes are produced. An impact evaluation is a form of outcome 
evaluation that assesses the net effect of a program by comparing program outcomes with 
an estimate of what would have happened in the absence of the program. This form of 
evaluation is employed when external factors are known to influence the program’s 
outcomes, in order to isolate the program’s contribution to achievement of its objectives. A 
process evaluation assesses the extent to which a program is operating as it was intended. 

It typically assesses program activities’ conformance to statutory and regulatory 
requirements, program design, and professional standards or customer expectations. 
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Agency Comments 


We provided a draft of this report and the electronic supplement to the 
Department of Education for review and comment. Education provided 
technical comments on the report, which we incorporated as appropriate. 
Education had no comments on the electronic supplement. 



We are sending copies of this report to appropriate congressional 
committees, the Secretary of Education, and other interested parties. In 
addition, the report will be available at no charge on GAO’s Web site at 
http://www.gao.gov. 

If you or your staff have any questions about the report, please contact me 
at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last page 
of this report. GAO staff that made major contributions to this report are 
listed in appendix II. 

George A. Scott 

Director, Education, Workforce, 
and Income Security Issues 



Page 25 



GAO-09-683 Career and Technical Education 



Appendix I: Scope and Methodology 



Survey of States 



To obtain national-level information on states’ implementation of Perkins 
IV, we designed and administered two Web-based surveys, at the 
secondary and postsecondary levels, to state directors of career and 
technical education (CTE) in the 50 states and the District of Columbia. 
The surveys were conducted between January and April 2009, with 100 
percent of state CTE directors responding to each survey. The surveys 
included questions about the types of data states collect for the student 
performance measures and challenges they face; the various kinds of 
technical assistance, guidance, and monitoring states received from 
Education; and how states evaluate their CTE programs. The surveys and 
a more complete tabulation of the results can be viewed at GAO-09-737SP. 

Because this was not a sample survey, there are no sampling errors. 
However, the practical difficulties of conducting any survey may introduce 
nonsampling errors, such as variations in how respondents interpret 
questions and their willingness to offer accurate responses. We took steps 
to minimize nonsampling errors, including pretesting draft survey 
instruments and using a Web-based administration system. Specifically, 
during survey development, we pretested draft instruments with officials 
in Minnesota, Washington state, and Vermont in December 2008. We also 
conducted expert reviews with officials from the National Association of 
State Directors of Career and Technical Education Consortium and MPR 
Associates, who provided comments on the survey. In the pretests and 
expert reviews, we were generally interested in the clarity of the questions 
and the flow and layout of the survey. For example, we wanted to ensure 
that terms used in the surveys were clear and known to the respondents, 
categories provided in closed-ended questions were complete and 
exclusive, and the ordering of survey sections and the questions within 
each section were appropriate. On the basis of the pretests and expert 
reviews, the Web instruments underwent some revisions. A second step 
we took to minimize nonsampling errors was using Web-based surveys. By 
allowing respondents to enter their responses directly into an electronic 
instrument, this method automatically created a record for each 
respondent in a data file and eliminated the need for and the errors 
associated with a manual data entry process. When the survey data were 
analyzed, a second, independent analyst checked all computer programs 
to further minimize error. 

While we did not fully validate all of the information that state officials 
reported through our surveys, we reviewed the survey responses overall to 
determine that they were complete and reasonable. We also validated 
select pieces of information by corroborating the information with other 
sources. For example, we compared select state responses with 
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Appendix I: Scope and Methodology 



information submitted to Education in state Perkins plans. On the basis of 
our checks, we believe our survey data are sufficiently reliable for the 
purposes of our work. 



Site Visits 



To better understand Perkins IV implementation at the state and local 
levels, we conducted site visits to three states — California, Minnesota, and 
Washington state — between September 2008 and February 2009. In each 
state we spoke with secondary and postsecondary officials at the state 
level with CTE and Perkins responsibilities. We also interviewed officials 
from local recipients of Perkins funds — that is, school districts and 
postsecondary institutions. Through our interviews with state and local 
officials, we collected information on efforts to implement the Perkins 
performance measures and uses of Perkins funding, experiences with 
Education’s monitoring and technical assistance, and methods for CTE 
program evaluation. The states we selected represent variation across 
characteristics such as the type of state agency (i.e., state educational 
agencies or state college and university systems) eligible to receive 
Perkins funds, the amount of Perkins IV funds received in fiscal year 2008, 
and type of approach used to measure student attainment of technical 
skills. The localities selected for site visits provided further variation in 
geographic location (urban versus rural), number of CTE students served, 
and amount of Perkins funding received. 

We conducted this performance audit from August 2008 to July 2009, in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Automated answering system: (800) 424-5454 or (202) 512-7470 


Congressional 

Relations 


Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 
U.S. Government Accountability Office, 441 G Street NW, Room 7125 
Washington, DC 20548 



Public Affairs 



Chuck Young, Managing Director, youngcl@gao.gov, (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548 
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